Behind The Demoscene Server
An insight on the demoscene's main storage facility
Credits:
Written by Melwyn & Magic
Thanks to Redhound for providing technical information and ftp stats
Photos taken by Redhound especially for this article
Prologue
The ftp.scene.org server is a huge part of today's demoscene. It's fair to say that
ftp.scene.org is even one of the main foundations of today's modern demoscene.
No need to swap disks anymore like in the old days, just download what you like! If you don't think about
it people just take it for granted almost every demoscene release is freely downloadable from the internet.
Most sceners have probably visited www.scene.org and made use of their server to download some releases.
But what is scene.org exactly? Who are the people who are making your download needs so much easier to
fullfill? This article will share some information with you to make you more aware about the existence of
scene.org, what scene.org actually is and who the people behind it are. This article is mostly written by
Melwyn of Farbrausch & Haujobb who originally was a webmaster for scene.org but later started to organize the
scene.org awards back in 2002, which has been his focus since then. Melwyn also initiated, some years ago,
the scene-id protocol, which is used for user authentication through demoscene portals, and he is doing
some web-programming every now and then.
Scene.org history and organisation
Scene.org started back in 1996 as a fm.org (five musicians) domain, using which they shared their music releases with
the friends. It expanded from there and around 97-98 it started providing party releases.
When Hornet was shut down (in 99), it became the major repository for scene releases, which it still is.
The founders were mellow-d and other members of the five musicians gang. The current staff members can be found here:
http://www.scene.org/staff.php. The main server used to be located in Helsinki, Finland at several different
places until 1999 when it moved to the Netherlands because of too much bandwidth usage. Currently it is located at
the Rotterdam Highschool where Redhound (the main ftp admin) is working. Behind all this is a Finnish-based
non-profit organization: the international scene organization ry. This organisation takes care of the
monetary issues, funding server maintenance, scene.org awards and other projects that
are hosted under scene.org. The members of this non-profit organization are also known as scene.org staff
members - mostly, at least. Scene.org staff members who are in this non-profit organisation are currently:
Esse r. (ior), Jesse h., Melwyn and Jeffry (redhound). That's just enough to have the required minimum by
the Finnish organizational law. But soon there will be a board meeting where this just mentioned list could
change. Scene.org is in good hands this way as nobody "owns" scene.org. You can say there's two parts of scene.org:
first the people who organize and maintain the archive, websites and the projects and then the physical layer - i.e. all
the servers and so on. In an unfortunate event of losing the main server, there are already enough mirrors that have
the complete compilation so that wouldn't be as bad a disaster as it sounds like. The same goes for losing people who are
doing all this. In fact new people are recruited when the old members get inactive and try to keep everything running
and hopefully doing even more for the scene.
Downtime, scene.org vs pouet.net
There have been some moments when the scene.org servers were offline, sometimes a whole weekend. Some sceners
can't handle this very well since pouet.net is thus offline as well. Not having the possibility to access pouet.net makes
some sceners react heavily on the other demoscene portals out there. Hugi learned that this could soon be a problem of
the past. There have been several failures with the file system where no data has been lost, but it seems when this
happens, the raid array becomes unreachable from the server and requires a reboot at place. If this happens during
the weekend, there's nothing anyone can do about it. There are plans to separate the www and the ftp server so at
least the main website and hosted websites such as pouet.net will be available even when the ftp server is down.
Pixar / Scene.org Awards
By the help of sponsors scene.org can exist. Recently Pixar (Toy Story, Finding Nemo etc.) donated a large
sum of money to scene.org. Pixar's interest began originally from the scene.org outreach done at the fmx film festival,
spring 2007 in Stuttgart, Germany. Since then scene.org has been in talks with Pixar and luckily they decided to sponsor
sceme.org. There's something to learn from the realtime department also for Pixar and other companies in the cgi industry.
For example demoing your animation without long precalculation or wait time for rendering can be very useful.
The scene.org awards is also an annual event held by scene.org. It has been held for 5 years now. Scene.org hasn't started yet with
the preparations for 2008 but if everything is right it will be about the same ceremony as the 2007 edition - no big
changes. Perhaps the organizing of the scene.org awards has gotten more professional even if it's still all done by
volunteers. Five years have passed since the first edition - perhaps it's the right time to look back at this after 10 years.
Scene.org technical information and ftp stats
Hugi contacted Redhound of scene.org who is the main system and ftp admin
for the most recent technical information and ftp stats. Here is a brief summary:
Scene.org server (since the 16th of June 2007):
HP Proliant DL380 G4
Dual Intel Xeon 3.2GHz
3 GB ram
Internal Storage:
Onboard HP Smart Array 5i raid controller
5x 72GB U320 SCSI disks inside a Raid5+hotspare setup (total 197GB)
External storage array (since the 18th of August 2006):
LSI Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI controller
Transtec/AXUS BR-1600U3-R IDE-to-Ultra160 external SCSI enclosure
16x 300GB PATA disks inside a Raid5+hotspare setup (total 4.5 terabyte)
This is the first server on the "enterprise" level. Before the current server
scene.org was run on a self built server. It was not very fast and not really
good for the serverload scene.org reaches. All current hardware is sponsored by
the Rotterdam Highschool. Redhound is working there with quite a bunch of Dutch
(ex-) sceners: Theperic, Elmar, Andabata, Gewse, Izard and Naiad.
Network:
1Gbit/s network interface (onboard Broadcom BCM5703X Gigabit Ethernet)
connected on the core switch from the Rotterdam Highschool (a Cisco
Catalyst 6513) connected to a 1Gbit/s uplink to SURFnet to reach the
internet. The Rotterdam Highschool is the primairy SURFnet POP for the
city Rotterdam and has 2x 10Gbit/s uplinks to the SURFnet core in Amsterdam.
Now we will give you, as an example, a total summary of the scene.org FTP stats
over the period 28th of July 2007 until the 25th of August 2007:
Files Transmitted During Summary Period: 1,609,730
Average Files Transmitted Daily: 59,619
Bytes Transmitted During Summary Period: 9,295,783,740,510
Average Bytes Transmitted Per File: 5,774,747
Average Bytes Transmitted Daily: 344,288,286,685
That's 9 Terabyte in total FTP traffic with an average of 344GB a day.
Redhound told Hugi that August is always a top month for scene.org.
Normally there is an average of 200GB a day.
Scene.org - New features on their way?
As to be read on different websites, scene.org could get some changes in the short term or at least in 2008.
Nothing is decided yet, there's some technical development happening under the hood,
but it's not very visible. For example work on the archive, developing internal tools for better maintenance and such. Hopefully in
2008 we'll finally have a better upload functionality for both for individuals and party organizers. We think that
would be a good improvement for Scene.org for sure. Like an alternative uploading method
to anonymous ftp, linked to scene-id accounts. The idea is that then scene.org knows
who's uploading and scene-id users can be linked for example to their group/netlabel/party directories and maintain the content directly.
Uploaded copyrighted content can
be traced. However, I think the original author is always the responsible one in the
end... it has happened before, for example party organizers may not know that some entry had copyrighted content until later on.
No matter what upload method is used the author of the demo / music is resposible for the copyrights, not the uploader.
Epilogue
Hugi would like to thank scene.org which we now like to call 'the heart of the demoscene' as it 'pumps' the flow of
productions through the Internet and is hosting websites like for example pouet.net. We have learned that the main
service of scene.org will always be to provide the archive for the demoscene and secondarily scene.org aims to promote the
demoscene globally. This means that, not counting the outreach efforts, thanks to scene.org a lot of non-sceners know
about the demoscene through the scene.org portal and/or downloading demos there or through the scene.org awards visibility.
We hope that through this article you have gotten a better insight on scene.org. Remember this when you are downloading
for example the winning demos from the latest party. Don't take downloading for granted. We end this article with the scene.org motto which has been for a long time now and which
is also similiar to the one Hugi uses:
Scene.org - from sceners to the sceners
Happy downloading!
Melwyn & Magic